FEAT AzureContentFilterScorer: Switch to async client and accept async auth providers by adrian-gavrila · Pull Request #1467 · Azure/PyRIT

adrian-gavrila · 2026-03-13T23:27:55Z

Description

Switches AzureContentFilterScorer from the synchronous ContentSafetyClient to the async version (azure.ai.contentsafety.aio), unblocking the event loop and accepting both sync and async token providers. This brings the scorer in line with PyRIT's async-first architecture and mirrors the pattern already used by OpenAIChatTarget.

Problem: The scorer used the synchronous ContentSafetyClient, blocking the event loop on every analyze_text() / analyze_image() call. It also explicitly rejected async token providers with a ValueError, forcing sync-only auth even though the rest of PyRIT works with async auth.

Changes (6 files, +149 / -61):

pyrit/auth/azure_auth.py — New AsyncTokenProviderCredential class: wraps sync or async callables into an AsyncTokenCredential for the Azure SDK async client. Needed because unlike OpenAI's SDK (which accepts bare callables), the Azure SDK requires a credential object implementing the full AsyncTokenCredential protocol (get_token, close, __aenter__, __aexit__). Async counterpart to the existing TokenProviderCredential.
pyrit/auth/__init__.py — Export AsyncTokenProviderCredential.
pyrit/score/float_scale/azure_content_filter_scorer.py — Switched to async ContentSafetyClient. Added _ensure_async_token_provider() (same pattern as openai_target.py) to wrap sync callables at init time. api_key now accepts str | Callable[[], str | Awaitable[str]]. Default (no key) falls back to Entra ID via get_azure_async_token_provider(). Removed sync rejection guards.
tests/unit/score/test_azure_content_filter.py — Updated mocks to AsyncMock, converted rejection tests to acceptance tests, added iscoroutinefunction assertions. 17/17 passing.
tests/integration/score/test_azure_content_filter_integration.py — Removed AZURE_CONTENT_SAFETY_API_KEY gate; tests now use Entra ID auth by default, only requiring AZURE_CONTENT_SAFETY_API_ENDPOINT. 2/2 passing.
tests/integration/mocks.py — Fixed pre-existing broken import (AttackIdentifier → ComponentIdentifier). This was blocking all 3 integration test files that import from this module.

Auth paths after this change:

`api_key` value	Behavior
`None` (no env var)	`get_azure_async_token_provider()` → async callable → `AsyncTokenProviderCredential` → async client
`"my-api-key"` (string)	`AzureKeyCredential(key)` → async client
Sync callable	`_ensure_async_token_provider()` wraps → `AsyncTokenProviderCredential` → async client
Async callable	Passes through → `AsyncTokenProviderCredential` → async client

Tests and Documentation

Unit tests (tests/unit/score/test_azure_content_filter.py): 17/17 passing. Updated 6 tests from MagicMock() to AsyncMock() for the now-async client methods. Converted 2 rejection tests into acceptance tests (async callables are now accepted, not rejected). Added inspect.iscoroutinefunction assertions to verify sync→async wrapping. Added pre-condition assertions on sync callable inputs.

Integration tests (tests/integration/score/test_azure_content_filter_integration.py): 2/2 passing. Tests now use Entra ID auth by default (the scorer's default when no API key is provided), requiring only AZURE_CONTENT_SAFETY_API_ENDPOINT and az login.

Pre-commit hooks: All passing

JupyText: Not applicable — no documentation notebooks were modified in this change.

adrian-gavrila · 2026-03-16T13:56:56Z

@microsoft-github-policy-service agree company="Microsoft"

Copilot

Pull request overview

This PR updates Azure Content Safety scoring to use the async Azure SDK client and expands authentication support so AzureContentFilterScorer can accept async token providers (and wrap sync providers for async compatibility).

Changes:

Switch AzureContentFilterScorer to azure.ai.contentsafety.aio.ContentSafetyClient and await analysis calls.
Add AsyncTokenProviderCredential and update scorer auth flow to support async (and wrapped sync) token providers.
Update unit/integration tests and integration mocks to align with async client and updated identifier typing.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`pyrit/score/float_scale/azure_content_filter_scorer.py`	Uses async Content Safety client; adds token-provider normalization/wrapping; awaits API calls.
`pyrit/auth/azure_auth.py`	Introduces `AsyncTokenProviderCredential` for Azure async client authentication.
`pyrit/auth/__init__.py`	Exports `AsyncTokenProviderCredential`.
`tests/unit/score/test_azure_content_filter.py`	Updates mocks to `AsyncMock`; revises token-provider acceptance tests.
`tests/integration/score/test_azure_content_filter_integration.py`	Moves credential/config skipping to module-level `pytestmark`.
`tests/integration/mocks.py`	Updates mock prompt target type signature to `ComponentIdentifier`.

You can also share your feedback on Copilot code review. Take the survey.

pyrit/score/float_scale/azure_content_filter_scorer.py

tests/unit/score/test_azure_content_filter.py

tests/integration/score/test_azure_content_filter_integration.py

…docstring fix - Add inspect.isawaitable() check in _ensure_async_token_provider wrapper to handle sync callables that return coroutines (e.g. lambda: async_fn()) - Add _returns_token tests that await scorer._api_key() and verify the actual token value for all three provider types (async, sync-returning- coroutine, sync) - Update integration test docstring to match skip condition: endpoint is required, API key is optional (Entra ID is default auth) - Remove unnecessary type: ignore comments flagged by mypy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR updates the Azure Content Safety scorer implementation and its test suite to use the async Azure SDK client and to support async (and sync) token-provider callables for authentication.

Changes:

Switched AzureContentFilterScorer to azure.ai.contentsafety.aio.ContentSafetyClient and awaited API calls.
Added support for async token providers (and wrapping sync providers for async compatibility).
Updated unit/integration tests and integration mocks to reflect the new async/auth behavior and corrected identifier typing.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`pyrit/score/float_scale/azure_content_filter_scorer.py`	Migrates to async Content Safety client, adds token-provider normalization/wrapping, awaits analyze calls.
`pyrit/auth/azure_auth.py`	Introduces `AsyncTokenProviderCredential` for async Azure SDK credential compatibility.
`pyrit/auth/__init__.py`	Exports `AsyncTokenProviderCredential`.
`tests/unit/score/test_azure_content_filter.py`	Updates mocks to `AsyncMock` and revises token-provider tests to accept async providers.
`tests/integration/score/test_azure_content_filter_integration.py`	Adjusts skipping logic and documentation to reflect Entra ID default auth support.
`tests/integration/mocks.py`	Updates `attack_identifier` typing to `ComponentIdentifier`.

Comments suppressed due to low confidence (1)

pyrit/score/float_scale/azure_content_filter_scorer.py:198

Since this scorer now uses azure.ai.contentsafety.aio.ContentSafetyClient, consider adding an explicit async cleanup path (e.g., a close_async() method) so callers/tests can deterministically close the underlying async transport/session and avoid resource warnings/leaks.

                credential = AsyncTokenProviderCredential(self._api_key)
                self._azure_cf_client = ContentSafetyClient(self._endpoint, credential=credential)
            else:
                # String API key
                self._azure_cf_client = ContentSafetyClient(self._endpoint, AzureKeyCredential(self._api_key))

You can also share your feedback on Copilot code review. Take the survey.

pyrit/score/float_scale/azure_content_filter_scorer.py

tests/integration/score/test_azure_content_filter_integration.py

tests/unit/score/test_azure_content_filter.py

pyrit/score/float_scale/azure_content_filter_scorer.py

The wrapping of sync providers is a normal init-time path, not worth INFO-level noise. DEBUG keeps the diagnostic available when needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

tests/integration/score/test_azure_content_filter_integration.py

pyrit/score/float_scale/azure_content_filter_scorer.py

- Remove module-level pytestmark skipif for AZURE_CONTENT_SAFETY_API_ENDPOINT - Add integration tests for image and text scoring with explicit API key auth - Use assert instead of pytest.skip so tests fail visibly when key is not set Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

pyrit/score/float_scale/azure_content_filter_scorer.py

Move _ensure_async_token_provider from openai_target.py and azure_content_filter_scorer.py into a shared public helper in pyrit/auth/azure_auth.py. Uses the more robust implementation (inspect.iscoroutinefunction + isawaitable handling) and logs at DEBUG level. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jsong468

great work, looks good to me! Just make sure the existing openai auth integration tests still pass :)

Adrian Gavrila added 2 commits March 13, 2026 16:41

Adding support for async in azure content filter scorer

b9f8edc

Fix pre-commit: add Returns docstring to __aenter__ and ruff format fix

f71b015

adrian-gavrila requested a review from Copilot March 16, 2026 14:10

Copilot started reviewing on behalf of adrian-gavrila March 16, 2026 14:10 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

pyrit/score/float_scale/azure_content_filter_scorer.py Outdated Show resolved Hide resolved

tests/unit/score/test_azure_content_filter.py Show resolved Hide resolved

tests/integration/score/test_azure_content_filter_integration.py Outdated Show resolved Hide resolved

adrian-gavrila requested a review from Copilot March 16, 2026 15:26

Copilot started reviewing on behalf of adrian-gavrila March 16, 2026 15:28 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

Adrian Gavrila and others added 2 commits March 16, 2026 11:47

Downgrade sync token provider log from INFO to DEBUG

ea6f548

The wrapping of sync providers is a normal init-time path, not worth INFO-level noise. DEBUG keeps the diagnostic available when needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into async-content-filter-scorer

2b46053

romanlutz approved these changes Mar 16, 2026

View reviewed changes

tests/integration/score/test_azure_content_filter_integration.py Outdated Show resolved Hide resolved

tests/integration/score/test_azure_content_filter_integration.py Show resolved Hide resolved

jsong468 reviewed Mar 16, 2026

View reviewed changes

pyrit/score/float_scale/azure_content_filter_scorer.py Outdated Show resolved Hide resolved

adrian-gavrila force-pushed the async-content-filter-scorer branch from c3fc0fa to 477132a Compare March 16, 2026 19:37

jsong468 reviewed Mar 16, 2026

View reviewed changes

pyrit/score/float_scale/azure_content_filter_scorer.py Show resolved Hide resolved

jsong468 approved these changes Mar 16, 2026

View reviewed changes

adrian-gavrila merged commit 8453bad into Azure:main Mar 17, 2026
38 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT AzureContentFilterScorer: Switch to async client and accept async auth providers#1467

FEAT AzureContentFilterScorer: Switch to async client and accept async auth providers#1467
adrian-gavrila merged 7 commits intoAzure:mainfrom
adrian-gavrila:async-content-filter-scorer

adrian-gavrila commented Mar 13, 2026 •

edited

Loading

Uh oh!

adrian-gavrila commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsong468 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

adrian-gavrila commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tests and Documentation

Uh oh!

adrian-gavrila commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsong468 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adrian-gavrila commented Mar 13, 2026 •

edited

Loading